Why is Evaluating Usability of Privacy Designs So Hard? Lessons Learned from a User Study of PRISM

Privacy is a thorny issue that affects all information systems designed for interpersonal interaction and awareness. Theoretical insights regarding privacy and user experience in a variety of systems have produced numerous design principles and guidelines for building systems sensitive to privacy issues. In order to truly improve support for privacy management, the usability of systems that implement these principles is critical. Yet, usability evaluation of privacy designs is a relatively unexplored area. In this paper, we describe our experience of conducting a longitudinal user study to evaluate the effectiveness of the privacy management enhancements offered by PRISM, a plugin for Instant Messaging (IM). Although the study was unable to achieve its intended objective, the lessons learned highlight the difficulties faced by evaluators of privacy management mechanisms. We hope that our experience will benefit future evaluations of privacy management mechanisms, and will initiate discussions on overcoming some of these challenges.